We introduce TestCase-Eval, a new benchmark for systematic evaluation of LLMs in test-case generation. TestCase-Eval includes 500 algorithm problems and 100,000 human-crafted solutions from the Codeforces platform. It focuses on two pivotal tasks: (1) Fault Coverage, which measures how well LLM-generated test sets probe diverse input scenarios and cover a wide range of potential failure modes. (2) Fault Exposure, which evaluates whether LLMs can craft a tailored test input that reveals a specific incorrect code implementation. We provide a comprehensive assessment of 19 state-of-the-art open-source and proprietary LLMs on TestCase-Eval, offering insights into their strengths and limitations in generating effective test cases for algorithm problems.

The recent program development industries have required problem-solving abilities for engineers, especially application developers. However, AI-based education systems to help solve computer algorithm problems have not yet attracted attention, while most big tech companies require the ability to solve algorithm problems including Google, Meta, and Amazon. The most useful guide to solving algorithm problems might be guessing the category (tag) of the facing problems. Therefore, our study addresses the task of predicting the algorithm tag as a useful tool for engineers and developers. Moreover, we also consider predicting the difficulty levels of algorithm problems, which can be used as useful guidance to calculate the required time to solve that problem. In this paper, we present a real-world algorithm problem multi-task dataset, AMT, by mainly collecting problem samples from the most famous and large competitive programming website Codeforces. To the best of our knowledge, our proposed dataset is the most large-scale dataset for predicting algorithm tags compared to previous studies. Moreover, our work is the first to address predicting the difficulty levels of algorithm problems. We present a deep learning-based novel method for simultaneously predicting algorithm tags and the difficulty levels of an algorithm problem given. All datasets and source codes are available at https://github.com/sronger/PSG_Predicting_Algorithm_Tags_and_Difficulty.

What are different types of Machine Learning and briefly explain them? The expected answer should mention supervised, unsupervised, and reinforcement learning. Supervised Learning You give the algorithm labeled data and the algorithm has to learn from it and figure out how to solve future similar problems. Think of it as if you're giving the algorithm problems and answers, the algorithm has to learn how these problems were solved in order to solve future problems in a similar manner. This is like the above example where the bank learns from your habits which credit card transactions are legit and which are fraudulent.

Machine Learning, AI, Deep Learning are buzz words being heard daily on TV, in workplaces, at gatherings, etc. Maybe you're a little bit embarrassed to ask what's Machine Learning or AI, or maybe you have the wrong understanding of Machine Learning. Either way that's okay because this article serves as an introduction to Machine Learning, I wrote it in a Q&A format so it becomes easy to follow and understand. Machine Learning is a subset of Artificial Intelligence (AI) and it's about writing software codes to enables computers (or machines in general) to get better at a given task on their own without human intervention. Some people argue that Machine Learning is a fancy way to say "Statistical Analysis" which is the science of collecting data and uncovering patterns and trends. Either way, think about all the data being generated daily and how people try to make sense of it to make their lives better, that's Machine Learning.